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Abstract 

This study aims to evaluate a professional development programme that prepares and assists teachers with the 
implementation of a multidisciplinary science module, basing the evaluation on participants ’ reactions, the first 
level of Guskey’s five-level model for evaluation (2002). Positive evaluations at the higher levels in Guskey's 
model are thought to depend upon those at the first level, that is, teachers' positive appreciation for the professional 
development programme. Different instruments (questionnaires, e-mail correspondence, and interviews) were 
used at various stages in the four-part professional development programme. The data were categorised by the 
stage in the programme to which they pertained: individual preparation; preparation seminar; online support; or 
reflection meeting. The data from all participating teachers were combined in word tables, with some parts 
paraphrased in a few sentences. 

In general, the teachers were satisfied with the professional development programme. This paper identifies several 
elements that ensure teacher satisfaction in the multidisciplinary professional development programme, indicates 
which are the difficult elements of this programme, and suggests how these difficulties can be handled in such a 
programme. 

Keywords: Professional development programme, evaluation, multidisciplinary science subject, innovation, 
teacher education 

1. Introduction 

The Ministry of Education, Culture, and Science in the Netherlands encourages multidisciplinary education. For 
their future careers, students must be able to work in multidisciplinary teams to solve interdisciplinary issues. 
Schools are therefore increasingly integrating multidisciplinary education into their programmes. In August of 
2007, the Minister of Education, Culture, and Science introduced an optional science subject called Nature, Life 
and Technology 1 (NLT) at the upper level of secondary education. NLT is a multidisciplinary subject, integrating 
elements from physics, chemistry, biology, mathematics, and physical geography. 

Teachers involved in the implementation of a curriculum innovation such as NLT, a new multidisciplinary subject, 
require new knowledge, skills and routines in order to teach the new subject adequately (Bergen & Van Veen, 
2004; Van den Akker, 1999). Over the years, research on effective professional development for teachers has 
shown that teachers can be prepared for a new curriculum through active involvement in the design and 
implementation of the curriculum innovation (Garet, Porter, Desimone, Birman, & Yoon, 2001; Keys & Bryan, 
2001; Loucks-Horsley, Love, Stiles, Mundry, & Hewson, 2003; Penuel, Fishman, Yamaguchi, & Gallagher, 
2007). 

Professional development becomes relevant and effective when long-term, collaborative, school-based approaches 
are used that focus on the learning of all students, that are linked to teachers' daily school practice, teachers’ prior 
knowledge, and the curricula that teachers must follow, and that also adjust to their diversity of behaviours and 


1 Also known as Advanced Science, Mathematics and Technology (ASMaT) 
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beliefs (Desimone, 2009; Garet et al., 2001; Hunzicker, 2011; Lieberman & Pointer Mace, 2010; Penuel et al., 
2007). In an earlier study, we used the research on professional development discussed above to design a generic 
model for a professional development programme for preparing teachers to teach a multidisciplinary science 
subject (Visser, Coenders, Terlouw, & Pieters, 2012). The aim of the current paper is to evaluate this previously 
designed and implemented professional development programme according to Guskey’s (2002) first level for 
evaluating professional development. A positive appreciation for Guskey’s the first level, participants’ reactions 
is necessary for positive results at the higher evaluation levels such as ‘teacher learning’ and, finally, ‘student 
learning outcomes’. Because the first level is the foundation of the other evaluation levels. In the conceptual 
framework section, we will explain in more detail the following: NLT as a multidisciplinary science subject; the 
concept of the professional development programme and how it has been instantiated in the particular professional 
development programme under evaluation; and the concept of evaluation of professional development. 

2. Conceptual Framework 

2.1 Nature, Life and Technology 

NLT is related to traditional science subjects but differs because of its multidisciplinary and modular structure. An 
NLT module consists of a situated practice (for example, nuclear fusion, sound design, or medicine) in which 
specific concepts traditionally belonging to physics, chemistry, biology, mathematics, and physical geography are 
explored. The NLT examination programme consists of six different domains; for each domain at least one of the 
three to eight existing modules must be taught. The modular and multidisciplinary character of NLT means that the 
content of the NLT modules amounts to more than just the sum of the contents of the traditional science subjects 
involved. Science teachers usually have a degree in one of the traditional science subjects. Therefore, teaching the 
NLT modules is challenging, because the modules not only deal with a teacher's own discipline, but also involve 
high-level content from other science disciplines. In order to encourage schools to offer high-quality education in 
this new subject, the National Steering Committee formulated a number of criteria that schools should meet to 
become an officially registered implementation school (Steering Committee NLT, 2007). The multidisciplinary 
nature of NLT requires that at least three teachers with different Master’s degrees (physics, chemistry, biology, 
mathematics, or physical geography) collaborate in a multidisciplinary teaching team. Such a multidisciplinary 
team has the freedom to select the modules for each examination domain, and can decide on the order in which the 
modules will be taught in their school. The school administration, in close consultation with the teaching team, 
determines which and how many teachers will be teaching a specific module. For instance, at one particular school, 
the module called ‘Medicines’ is taught by a team of two teachers, one a biology teacher and one a chemistry 
teacher. At another school, they decided to let one biology teacher rather than a team of teachers teach the module 
called ‘Cardiovascular system’ (a different module than ‘Medicines’ but in the same examination domain) to the 
same class level. Implementation varies from school to school as different choices are made. 

Implementation of a new subject with a complex and multidisciplinary nature such as NLT can therefore be 
regarded a curriculum innovation, in particular with regard to NLT’s structure, content, and delivery. 

2.2 Professional Development Programme 

Guskey (2000) defined professional development as “those processes and activities designed to enhance the 
professional knowledge, skills and attitudes of educators so that they might, in turn, improve students’ learning” (p. 
16). According to this author, three important factors influencing the quality of professional development are 
context, process, and content. 

In an earlier study we designed a generic model for a professional development programme to prepare teachers and 
assist them during the implementation of new multidisciplinary science curriculum on a multidisciplinary science 
subject (Visser et al., 2012). For this model, we combined the three factors Guskey described with three aspects 
important for the design of such a programme: multidisciplinary science features including school practices; the 
curriculum design phases; and professional development characteristics (Visser, Coenders, Terlouw, & Pieters, 
2010). These aspects and factors were translated into a generic model for a professional development programme 
for a multidisciplinary science subject, shown in Figure 1. The model designed can be used for all the NLT 
modules. 
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Programme stages: 

1 individual preparation 

2 preparation seminar 

3 online support 

4 reflection meeting 

Figure 1. Generic model for professional development programme (Visser et al., 2012) 

This professional development programme assists and supports teachers before, during, and after implementation 
of the new curriculum in a multidisciplinary science module. As mentioned before, three factors are relevant in this 
respect: context, process, and content. The context of the designed professional development programme in this 
case is the implementation of NLT within specific school programmes. The process consists of three phases. The 
first phase is scheduled before the actual implementation of a NLT module. The second phase takes place during 
implementation of the NLT module. The third phase occurs after the NLT module has been taught. The 
characteristics used to design the professional development programme are also the foundation for the content, that 
is, the stages of which the professional development programme is composed (Visser et al., 2012). The content of 
the professional development programme is specified as the four stages identified by the numbers 1 to 4 in Figure 
1. Each of these stages addresses particular types of issues. For instance, the issue of ‘the possibility of changing 
parts of the subject matter’, is tackled in the preparation seminar in the before teaching phase, as it is important at 
that point for teachers to exchange ideas about possible changes to be made. 

Over the years, much research has been done about the content of professional development programmes. For each 
stage (numbers 1 to 4 in Figure 1), we explain below first, what the literature has to say about this element of 
professional development programmes, followed by how this was addressed in our designed professional 
development programme for a multidisciplinary science subject. 

The participants in the professional development programme are: a coordinator to chair the professional 
development programme, an expert of the content of the module, a representative from the regional NLT 
information centre who knows about the possibilities for organizing field trips, the teachers who will teach the 
same module from different schools, and the school lab assistants. 

The before-teaching phase consists of an individual preparation part and a collaborative preparation part. 
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Individual preparation (Figure 1, number 1). 

Teachers who are implementing a curriculum innovation will go through a process of change. The Concerns Based 
Adoption Model (CBAM) by Hall and Loucks (1978) is considered a valuable model to describe what individual 
teachers go through during the process of innovation. In the initial phase of a professional development 
programme, teachers must therefore become aware of all the concerns in the process of innovation. Also a 
professional development programme is more successful when it connects everyday school practice and teachers’ 
prior knowledge, and also adjusts to the diversity of behaviours and beliefs of the participants (Cotton, 2006; Luft, 
2001 ). 

In the designed model for a professional development programme, the individual preparation starts a few weeks 
before teaching. The coordinator asks the teachers (via e-mail) to look over the upcoming module and consider 
three questions: 1) What knowledge and skills do I want my students to acquire in this module? 2) What kind of 
assessment methods and instruments do I intend to use? 3) What questions do I have for the preparation seminar? 
As it can take a long time for equipment for practical activities to be delivered, teachers are advised to find out 
what science equipment their school already has and what needs to be purchased or borrowed. 

Preparation seminar (Figure 1, number 2). 

Professional development is most relevant when it focuses on teachers’ existing needs; teachers’ personal needs 
therefore are the starting point for the preparation seminar (Dunne, 2002; Erickson, Brandes, Mitchell, & Mitchell, 
2005; Lieberman & Pointer Mace, 2008). 

Incorporating teachers’ own input in their interactions with colleagues (Day, 1999; Garet, Birman, Porter, 
Desimone, & Herman, 1999) creates a strong sense of ownership (Ogborn, 2002; Van den Akker, 1998; Vermunt, 
2006; Wikeley, 2005) and encourages active learning. Collaborative involvement with teachers of different 
schools in an interactive process can create a network, where teachers support and inspire each other and 
professionalise themselves (Clement & Vandenberghe, 2000; Jones, Asensio, & Goodyear, 2000). 

In the designed model for a professional development programme the preparation seminar is held with all 
stakeholders, two to three weeks before teaching. An overview of the outline of the preparation seminar is depicted 
in Table 1. 


Table 1. Collaborative seminar programme including underlying goals and teacher activities 


Outline of the 

Collaborative Seminar Programme Goal What teachers do 


I. 

Opening 



II. 

Soliciting teachers’ questions to be 
included in the programme 

To create strong sense of ownership 
with the programme 

Mention personal questions 

III. 

Discussion of the overall module 

To familiarize teachers with 
alternative ideas, and to form a clear 
picture of the overall module 

Discuss and exchange 
experiences, ideas, and good 
practices 

IV. 

Substantive aspects of the module 

• Experiments 

To handle practicalities, and discuss 
solutions for practical problems 

Try out experiments from the 
module 


• Module outline 

To prepare module and lesson outline 

Outline the module for own class 

use 


• Design materials 

To develop additional learning 
materials 

Discuss additional learning 
materials, assessments methods 
and instruments 

V. 

Appointments 

To organise support during teaching 
and organise field trips and guest 
lectures 

Reflect on the professional 
development programme and 
make appointments 

VI. 

Closure 
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After arrival at the seminar, teachers' questions from their individual preparation activity are collected, categorised 
and included in the seminar programme (Table 1, II). The questions can be related to content, planning, or 
equipment. Thereafter, teachers get an overview of the structure and operationalisations of the entire module 
(Table 1, III), followed by a more in-depth approach to some substantive aspects of the module, such trying out 
experiments, designing a module outline for use in their own class, and developing additional (learning) materials 
(Table 1, IV). The breadth and depth of these substantive aspects depend on the topic and content of the module. 
Through discussion and exchange of experiences, ideas, and good practices, participants tackle issues related to 
these substantive aspects. Reflection on the programme and making appointments for the next phase conclude the 
seminar (Table 1, V). The seminar is a first step to the creation of a collaborative network among teachers from 
different schools who will be teaching the same module. 

Online support (Figure 1, number 3). 

Teachers need support during the teaching phase, especially when they need to implement new approaches in their 
classes (Dede, Ketelhut, Whitehouse, Breit, & McCloskey, 2009). Due to time constraints, organising face-to-face 
meetings during the teaching phase is complicated; however, an online component can provide such support 
(Berger, Eylon, & Bagno, 2008; Owston, Sinclair, & Wideman, 2006, April). To support teachers during this 
teaching phase, e-mail exchange can be selected as a tool enabling all teachers to participate in the online exchange 
(Berger et ah, 2008). Owsten (2006, April) suggests that short weekly postings of teachers’ reflections might 
stimulate others to participate in the exchange. 

In the designed model for a professional development programme, the coordinator organises collaborative online 
support for all teachers, which continues for the entire teaching period. E-mail exchange is chosen as enabling all 
teachers to participate in the online exchange. During the teaching phase of the NLT module, on each Friday 
teachers receive an e-mail from the coordinator requesting them to describe their individual good practices and 
problematic experiences with the module for that week. They can also post personal requests for assistance or 
guidance on aspects of the module in this e-mail, called the ‘Friday e-mail’. All responses are summarized by the 
coordinator and returned to all participating teachers a few days later. The coordinator ensures that all teachers’ 
questions are addressed. 

Reflection meeting (Figure 1, number 4). 

Teachers’ reflection about their own practice is important for their professional development and facilitates 
cooperation among colleagues (Margalef Garcia & Pareja Roblin, 2008). Joint reflection is an important learning 
activity (Meirink, 2007), as strengths and weaknesses of the teaching-learning process emerge and are described 
for future use. 

In the designed model for a professional development programme teachers meet for collaborative reflection when 
the NLT module is finished and when teachers have had enough time to assess student learning outcomes. During 
the reflection meeting, issues related to the NLT module are discussed, such as strengths and weaknesses, learning 
goals, the module outline, teaching methods, and assessment methods. Through such a collaborative process, 
teachers can build up confidence for the next module to be taught. 

2.3 Evaluation 

Professional development evaluation is an essential component of programme improvement and renewal and 
long-term success (Rovai, 2003). 

Guskey (2002) developed a five-level model for evaluating professional development. The levels in this model are 
hierarchically arranged from simple to complex, with each higher level building on the ones before it. The more 
basic level of positive appreciation for the professional development programme is therefore necessary for positive 
results at the higher evaluation levels such as ‘teacher learning’ and, finally, ‘student learning outcomes’. Given 
this dependency of higher levels on those coming before, in this article, the focus is on Guskey's first level, 
participants’ reactions. Things to consider at this level according to Guskey are ‘basic human needs’ such as the 
quality of food and comfort of the room, and whether participants ‘liked’ the experience, whether the materials and 
presentation ‘make sense’ and whether presenters seem ‘knowledgeable and helpful’. Because the first level is the 
foundation of the other evaluation levels, the professional development programme must be serviceable for all 
participants. Participants’ evaluative reactions are described in this study as a measure of ‘satisfaction’. 
Satisfaction is defined as the following aspects: awareness of concerns, addressing these concerns, contributing 
ideas, usefulness, and creating self-confidence. The evaluation is positive when teachers are not dissatisfied. 

If the professional development programme fails to satisfy participants’ needs, a determination should be made as 
to whether this is due to the design or the delivery of the professional development programme. 
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3. Research Question 

This study aims to use Guskey's first level to evaluate the professional development programme that prepares and 
assists teachers with implementation of the new curriculum for the multidisciplinary science subject of NLT. The 
general research question is ‘How do participating teachers evaluate the professional development programme in 
terms of satisfaction?’ There are four specific sub-questions: (1) How do participating teachers evaluate the 
individual preparation? (2) How do participating teachers evaluate the preparation seminar? (3) How do 
participating teachers evaluate the online support? (4) How do participating teachers evaluate the reflection 
meeting? 

4. Method 

4.1 Context 

The professional development programme was carried out twice, for two different NLT modules. Because at least 
one module from each NLT examination domain has to be taught at each school ( Steering Committee NLT, 2007), 
we chose modules from two different NLT examination domains. The first is more physics oriented, while in the 
second module the focus is more on biology. Both modules are new and written for upper secondary school level. 
The first professional development programme, for the module ‘The hydrogen car’, ran from August 2009 till 
November 2009. The second professional development programme, accompanying the module ‘The brain and 
learning’, began in October 2009 and ran up to January 2010. 

4.2 Participants 

A written invitation to participate in this professional development programme was sent to thirteen teachers at 
thirteen different officially registered NLT implementation schools in the eastern part of the Netherlands. Five 
schools did not respond to the invitation and three schools were excluded from the professional development 
programme. These three schools were not yet teaching NLT at the upper secondary school level because of a later 
start of implementation. One school was ready to participate in the professional development programme but 
withdrew because the teacher changed jobs. A group of six teachers from four different secondary schools 
participated in the first professional development programme on the module ‘The hydrogen car’. Three 
participants were chemistry teachers (William, Ben, and Daniel), two were physics teachers (Adam and Jack), and 
one was a mathematics teacher (Oliver). All were male and all had more than nine years of teaching experience. 
They had all taught at least one NLT module before; however , the module ‘The hydrogen car’ was new for all 
teachers. For more information about the multidisciplinary module ‘The hydrogen car’ is presented in appendix 1. 
In two schools this module was taught by teams of two teachers. The two teaching teams both consisted of one 
chemistry teacher and one physics teacher. 

Five different teachers from the same four schools with the same four classes of students participated in the second 
professional development programme in order to prepare for the second module on ‘The brain and learning’. Three 
participants were biology teachers (Emma, Sophie, and Luke), one was a chemistry teacher (Max), and one was a 
physics teacher (Owen). Three were male and two were female, and all had more than nine years teaching 
experience. They had all taught at least one NLT module before; however, the module ‘The brain and learning’ 
was new for all teachers. At one school the module was taught by a team of two teachers, one chemistry and one 
biology teacher. 

4.3 Data Instruments and Analysis 

To answer the research questions, different instruments were used at various stages in the professional 
development programme. Right after the preparation seminar (Figure 1, number 2), each teacher filled in a 
questionnaire (A). During teaching, all the (Friday) e-mail correspondence as part of the online support (Figure 1, 
number 3) was collected (B). Before the reflection meeting (Figure 1, number 4), another questionnaire was sent 
(C). Shortly after the reflection meeting, an interview was held with each of the teachers (D). All the interviews 
were recorded and transcribed. Table 2 shows which instruments were used in which part of the professional 
development programme. 
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Table 2. Overview of the instruments used in the four stages of the professional development programme 


Instruments —> 

A 

questionnaire 

B 

e-mails from online support 

c 

questionnaire 

D 

interview 

1. Individual preparation 




X 

2. Preparation seminar 

X 


X 

X 

3. Online support 


X 

X 


4. Reflection meeting 



X 

X 


The data were organized by the four stages of the professional development programme to which they pertained: 
individual preparation, preparation seminar, online support, or reflection meeting. The data from all teachers 
across both professional development programme episodes were combined in word tables, and some parts of these 
tables were paraphrased in a few sentences. For details of how the instruments are used in the four stages see 
appendix II. 

5. Results 

In the next section, the findings for both professional development programmes will be presented according to the 
four research subquestions. 

Reactions on the individual preparation (1) 

Two questions were posed to evaluate the individual preparation. With regard to the first question, teachers from 
both modules agreed that it is important to start early with preparation of the module. As the subject and modules 
are new (teachers must acquire new knowledge, skills, and routines) this importance is well founded, because 
normally preparation took place just before teaching, as teachers said, ‘due to school constraints’. According to all 
teachers, the e-mail from the coordinator helped to start the preparation earlier. With regard to the second question, 
all the teachers reported that the individual preparation was also helpful for checking and ordering the science 
equipment needed for the module. 

‘What questions do I have for the preparation seminar?’ could have been related to content, planning or equipment. 
The eleven teachers of modules I and II formulated thirteen content questions, twenty planning questions and four 
questions about equipment. It appeared that Max and Luke, teachers of module II (The brain and learning), did not 
carry out the individual preparation activity adequately. Max’s inadequate preparation was due to his insignificant 
role in the planned teaching of the class, which was to be lead by his co-teacher Sophie. Luke fell short in 
preparation because of a later time deadline. The defined time period for the preparation seminar was two to three 
weeks before teaching the module. For Luke, this was impossible, as he would start teaching the module nine 
weeks later. Therefore, Luke was at the preparation seminar but did not have any questions he would have liked to 
have answered in the preparation seminar. Ben’s statement exemplifies the way the teachers responded in 
generating questions: ‘I would like to receive an overview of the whole module (planning), the ins and outs about 
the experiments of the module (content, planning, and equipment) and discuss the content of the assessment 
method (content)’. 

Reactions on the preparation seminar (2) 

Teachers answered some open questions to evaluate the preparation seminar. 

When asking ‘What was missing from the preparation seminar?’, ten of eleven teachers did not think anything was 
missing. Only Emma said, ‘I missed concrete information like telephone numbers for field trips and guest 
lecturers’. In the final interview the teachers were asked the same question again. This time seven of the eleven 
teachers did not think anything was missing. Three teachers of module I missed ‘trying out the complete 
experiments of the module’, and Max said, ‘I’d have liked some more information on different teaching methods 
like movies, etc.’. 

The teachers responded to open questions about the contribution of the preparation seminar to different categories 
of ideas, and their responses are described below, with some teacher statements included to illustrate the general 
tenor of the answers. A teacher could provide multiple answers. The number of teachers who responded similarly 
is shown in parentheses. 

a) All eleven teachers from modules I and II responded that the preparation seminar contributed to ideas for 
preparing the experiments. Particular responses included: ‘where to purchase necessary equipment’ (4), ‘all the 
materials for the experiments were shown’ (3), and ‘talked about supplementary experiments and research’ (1). 
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William said, ‘I received practical tips and good support for the lab assistant’. Two others responded in a similar 
manner. 

b) Nine out of eleven teachers from modules I and II indicated that the preparation seminar contributed to ideas 
for the planning and organisation of the module. Among the responses were: ‘gained ideas to teach the module, 
lesson plan’ (9), ‘ideas for assessment methods’ (3), ‘suggestions for teaching the sequence of the chapters, parts 
students study individually and in a group’ (3) and ‘materials that can be used’ (1). Ben said, ‘We discussed 
different possibilities for teaching the module; now I can quickly design a lesson plan at home’. 

c) All six teachers from module I indicated that the preparation seminar contributed to ideas for different 
assessment methods. Particular responses given were: ‘using portfolios’ (3), ‘theoretical test’ (2), ‘presentations’ 
(2), and ‘assess groups of students’ (2). Adam responded: ‘I gained lots of ideas, for instance, a different way to do 
a theoretical test’. 

d) Four out of five teachers from module II indicated that the preparation seminar contributed to ideas for field 
trips or guest lecturers. Their responses included: ‘gained ideas’ (4), and ‘explicit contacts and appointments must 
be made’ (2). Luke said: ‘I have to read a bit more of the module, but I probably won’t organise a field trip or a 
guest lecture’. 

e) All five teachers from module II indicated that the preparation seminar contributed to ideas for motivating 
students. Responses that were made: ‘motivating films’ (3), ‘change chapter sequence’ (2), ‘themes for students' 
own research projects’ (1), ‘a catchy introduction to the module’ (1), and ‘experiments and field trips’ (1). Max 
said: ‘From now on, the starting point of a module is motivating students’. 

All six teachers from module I and three out of five teachers from module II (Luke, Sophie and Max) responded to 
similar questions presented in a Likert-scale format, as described above. The Likert scale was from 1 to 5 (5 = 
excellent). Table 3 shows the number of participants rating each question, and includes the mean scores. 


Table 3. Number of participants rating each question on a 1-5 Likert scale about the contribution of the preparation 
seminar to ideas (Instrument C) 


Likert scale score —* 

1 

2 

3 

4 

5 

Mean 

sd 

Has the preparation seminar contributed ideas ... 

a)... for preparing the experiments? 


3 

2 

3 

1 

3.2 

1.1 

b)... for the planning and organisation of the module? 

- 

- 

1 

2 

6 

4.6 

0.7 

c)... for different assessment methods? 

- 

- 

2 

6 

1 

3.9 

0.6 

d)... for field trips or guest lecturers? 

5 

1 

2 

1 

- 

1.9 

1.2 

e)... for motivating students? 

- 

- 

4 

5 

- 

3.6 

0.5 

1) ... about the chapter order in your planning? 

- 

- 

2 

2 

5 

4.3 

0.9 

g)... how to deal with the prior knowledge of students? 

1 

2 

2 

4 

- 

3.0 

1.1 

Mean = Mean score 1 = weak 

sd = Standard deviation 5 = excellent 









The reliability (internal consistency) of the teachers’ answers was calculated, resulting in a Cronbach’s alpha of 
0.93. Taking into account the norm that Cronbach’s alpha should be larger than 0.80 we consider this questionnaire 
to be reliable (Heus De, Leeden Van der, & Ganzendam, 1995) 

Additional open questions were posed immediately after the preparation seminar. One was, ‘How useful was 
attending the preparation seminar?’. All eleven teachers perceived the preparation seminar to be useful. Their 
responses included that it was ‘a well prepared day’ (by 3 teachers), it had ‘an open work atmosphere’ (6), ‘in a 
nice room’ (1), with ‘good food’ (5). The seminar was ‘motivating’ (4), ‘effective’ (4) and gave a ‘good overview 
of the module’ (2). Sophie said: ‘The seminar was effective, I now have a good overview of the entire module 
including the assessment methods and the organisation’. 

Teachers were also asked, ‘How has your understanding of teaching changed after the preparation seminar?’ and 
‘How has your self-confidence changed after the preparation seminar?’ After the preparation seminar, all the 
teachers said they had a better understanding of teaching the module than before. Ten out of eleven teachers had 
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more confidence in teaching the module after the preparation seminar, because of ‘more background knowledge’ 
(4), ‘a better overview of the module’ (4) and ‘gained lots of ideas for teaching the module’ (5). One teacher, Ben, 
said: ‘I did not gain more confidence; I will manage if. 

Reactions on the online support (3) 

An overview of the e-mail traffic during the teaching weeks is depicted in Table 4. The numbers of lessons per 
week and the amount of teaching weeks per NLT module differ per school. The school administration, in close 
consultation with the teaching team, makes different choices; therefore, implementation varies from school to 
school. 


Table 4. Overview of teaching information and number of responses to the coordinator on the Friday e-mail 



School 

Lessons 

Teaching 

Responses in 

Percentage of 

Total 


(teacher) 

per week 

weeks 

Friday e-mail 

response 

requests 


1 (William) 

3 (60 min) 

6 

6 

100% 

2 


2 (Ben and Adam) 

3 (50 min) 

8 

5 

63% 

0 

"3 

3 (Daniel and Jack) 

3 (50 min) 

8 

7 

88% 

1 

o 

s 

4 (Oliver) 

2 (50 min) 

11 

6 

55% 

0 


1 (Owen) 

3 (60 min) 

6 

6 

100% 

0 

- 

2 (Sophie and Max) 

3 (50 min) 

8 

7 

88% 

1 

"3 

3 (Emma) 

3 (50 min) 

8 

0 

0% 

0 

o 

s 

4 (Luke) 

2 (50 min) 

11 

2 

18% 

0 


Luke sent only two e-mails, as a result of the later period of teaching the module in his school, which left an 
overlap of two weeks with the other schools. Due to a lack of time, Emma withdrew from the professional 
development programme after the preparation seminar. In total, there were four requests during teaching. William 
had two requests. First, he would like to organise an field trip at the university, but the contact person he met at the 
preparation seminar did not respond. He asked the coordinator to help him to contact this person. Second, he asked 
all participating teachers of module I for help about marking the students’ exercises: ‘What is a good assessment 
method or instalment for students’ exercises?’ Jack requested whether someone could send him the answers of the 
exercises, because he had lost them. Sophie requested a copy of a movie mentioned in the module, that introduced 
a particular theoretical component, for her school. 

The teachers were asked how useful the Friday e-mail exchange was and whether they would prefer a different 
medium of exchange. Five out of eleven teachers found it useful. They said: ‘useful tips’ (by 5 teachers), 
‘notification of similar problems that I experience’ (3), ‘exchange and tips of exercises for tests’ (2). Daniel found 
it not so useful, he said: ‘I got very little from the Friday e-mail exchange. The Friday e-mail request came 
sometimes at a busy time’. Seven out of eleven teachers did not prefer a different medium of exchange. Three 
teachers preferred a collaborative meeting during the teaching phase, and one teacher (Max) did not answer this 
question. Sophie said: ‘I aspect a higher output, including material exchange with a collaborative meeting instead 
of Friday e-mail exchange’. 

Reactions on the reflection meeting (4) 

Four of the six teachers of module I were present at the reflection meeting. Adam and Oliver were absent due to 
obligations elsewhere. The reflection meeting after module II was cancelled because of a variety of circumstances. 
Emma withdrew from the professional development programme after the preparation seminar. Owen could not 
participate in the reflection meeting because of illness during the period when the reflection meeting was being 
organised. Besides that, because of the differences in teaching weeks between schools, there was not a good point 
at which to organise the reflection meeting. Max and Sophie had finished teaching, while Luke was half way 
through teaching the module, because of his later start as mentioned before. 

Table 5 shows the three questions and short answers about the reflection meeting. 
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Table 5. Questions and answers concerning satisfaction regarding the reflection meeting from four teachers 
participating in the professional development programme of module I 

Module I 

4 teachers participating 

How do you assess the 
collaborative reflection 
meeting? 

- useful (3) 

- pleasant end to the professional development programmed) 

- exchange of information about how other schools implemented the module (1) 

- ideas for improvement for next year (1) 

What are useful elements from 
discussing the module during 
the evaluation meeting? 

- ideas for improvement for next year (3) 

- other teachers confirmed my own thoughts (2) 

What are useful elements from 
the discussion of the points 
contributed? 

- each school has similar problems (2) 

- ideas for dealing with small numbers of students in the class (1) 

- ideas for scheduling NLT (1) 

- ideas for informing grade three students about taking NLT (1) 

- talking with colleagues about all aspects of NLT (1) 


The collaboration with colleagues from different schools was assessed positively by all eleven participating 
teachers. When asked what was positive about the collaboration with colleagues from other schools, they said: ‘the 
exchange of ideas and good practices’ (by 6 teachers). Teaming from each other’ (3), ‘dealing with similar issues’ 
(3), and it was ‘inspiring’ (1). A teacher could give multiple different answers. William said: ‘the exchange of 
ideas with colleagues is pleasant and inspiring’. 

6. Conclusions and Discussion 

The focus of this paper is on Guskey’s (2002) first evaluation level, participants’ reactions. A positive 
appreciation for the professional development programme is thought to be necessary for positive results at the 
higher levels of Guskey’s five-level model for evaluation. The general research question in this study was as 
follows: ‘How do participating teachers evaluate the professional development programme in terms of 
satisfaction?’. Overall, the participating teachers positively appreciated this professional development programme 
designed to assist and support teachers before, during and after implementation of a multidisciplinary science 
module. The teachers were generally satisfied with the professional development programme. Several elements of 
the programme ensured teacher satisfaction. During the individual preparation the teachers became acquainted 
with the module, became aware of key aspects of the module, and formulated questions about elements they 
considered problematic. The preparation seminar addressed these concerns, contributed to their acquired new 
ideas, was useful for motivation and implementation in class, and created self-confidence. The online support 
addressed concerns that emerged during teaching, and the final reflection meeting provided ideas for improvement 
for next year. 

Literature shows that professional development is more effective when it is long-term (Desimone, 2009; Garet et 
ah, 2001; Hunzicker, 2011; Lieberman & Pointer Mace, 2010; Penuel et ah, 2007). However, this study shows that 
organising a long-term professional development programme for NLT teachers is a challenging endeavour, 
because of the following three specific characteristics of NLT: 

1. NLT is taught by one or more teachers from different science disciplines. Practically, this entails that when a 
module has a strong physics component, a physics teacher will be the responsible teacher or will be a member of 
the teaching team. When the next module is biology-oriented, a biology teacher will be in charge. A professional 
development programme to prepare and assist teachers in the first example will mainly have physics teachers 
participating, whereas for the second module it will be mainly biology teachers who enter the programme. At the 
individual teacher level, this inhibits a long-term continuous professional development programme. In this study, 
teachers from four secondary schools attended the two professional development programmes. At these four 
schools, the administration (in close consultation with the teaching team) decided to let other teachers teach the 
second module. Therefore, the teachers in charge of module I were all different from the teachers of module II. On 
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top of this, the number of teachers responsible for a module varied: the first module was taught by six teachers, two 
single teachers and two pairs, whereas the second module had five teachers, three single teachers and one pair. 

2. The multidisciplinary teaching team also has the freedom to select the modules for each examination domain, 
and can decide the order in which the modules will be taught in their school. In practice, this implies that the same 
class level in different schools are taught different modules or the same modules in a different order. At the 
individual teacher level, this obstructs the continuity of a long-term professional development programme. 
However, even if schools choose to teach the same modules and let them be taught by the same teachers, the 
characteristic described in the next paragraph remains an obstacle for the organisation of the professional 
development programme. 

3. The school administration has some freedom to determine the lesson timetable. In practice this means that 
schools differ in the number of lessons per week, and therefore in the number of teaching weeks per module. This 
causes asynchronous teaching periods across schools, interfering with the organisation of the individual 
preparation before teaching and the effectiveness of the collaboration in the online support during teaching. In this 
study, little overlap (20%) existed between the teaching weeks of module II at the four schools. When the teaching 
periods for the module overlapped by about 75%, as was the case for module I, teachers at schools prepared 
individually, and had similar experiences so that online support could facilitate collaborative exchange. Possible 
solutions for emerging problems were discussed. This encourages active learning and can ultimately lead to the 
formation of a support network (Clement & Vandenberghe, 2000; Garet et al., 1999). We therefore conclude that 
when organising a professional development programme, attention should be paid to the defined period between 
the individual preparation, the preparation seminar, and the start of teaching the module. It is important that the 
time between these phases is not too long, so that the professional development programme will be connected to 
teachers’ daily school practice and the curricula that teachers must follow. In this way the professional 
development becomes more relevant and effective (Desimone, 2009; Dunne, 2002; Garet et ah, 2001; Hunzicker, 
2011; Lieberman & Pointer Mace, 2010; Penuel et ah, 2007). 

In this study, we chose e-mail as a simple tool to enable all teachers to participate in the online support during 
teaching. When teachers implement new approaches in their classes, support during teaching is important but 
complicated (Dede et ah, 2009). Because of teachers’ heavy daily work load, they lack time for online interactions; 
this often results in a poor participation rate (Berger et ah, 2008; Owston et ah, 2006, April). In order to improve 
the participation rate, the coordinator of the professional development programme in this study took the weekly 
initiative of sending every participant an e-mail on Friday (Visser et ah, 2012). Due to the coordinator's initiative, 
the number of responses from the teachers for these so called Friday e-mail was relative high, five to seven mails in 
the six to eleven weeks of teaching (55% - 100% responses on the e-mails). The Friday e-mail seemed to be a 
promising medium of exchange, but the content of the correspondence was disappointing. Teachers mainly posted 
information about the subject matter they taught, what students did during the lessons the past week, and what they 
were planning to do the following week. Three out of eleven teachers made four requests. Only one of these 
requests was relevant for all participants: ‘What is a good assessment method or instrument for students’ 
exercises?’. The coordinator could help with the other three requests: ‘Help with contacting a person from the 
university, receiving the answers of the exercises, and receiving a copy of a movie’. Five out of eleven teachers 
assessed the Friday-email as useful. The teachers shared their experiences of similar problems, exchanged 
additional materials, and learned useful tips for teaching the module. Three out of eleven teachers preferred a 
different medium of exchange during teaching, for example, a face-to-face meeting. The weekly Friday e-mail 
exchange is time consuming. The mail arrives at a busy moment, and teachers need discipline to respond every 
week. Therefore, in the organisation of a professional development programme, it is advisable to reduce the 
frequency of responses to once a fortnight instead of once a week, to maintain continuity between the meetings, 
with the possibility of sending a personal request when needed. When participating teachers prefer a collaborative 
face-to-face meeting during the teaching period, and time constraints allow it, this can additionally be arranged. 

7. Implications 

Teaching NLT is complex as it requires one or more teachers from different science disciplines to co-teach the 
modules. Another feature demonstrating its unique character is that the teaching team has to select the modules, the 
order in which these will be taught, and the timetable to be used. These complex environment makes it a challenge 
to develop a professional development programme that satisfies all participating teachers. This paper evaluate the 
important first level, because this level is the foundation for the other higher evaluation levels. Teachers learn most 
when they appreciate the professional development programme. Subsequent research will explore Guskey’s level 
two to five respectively: what participants learned from the professional development programme, if 
organisational factors hinder the success of the professional development programme, whether participants used 
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new knowledge and skills in class, and how it affected students’ learning outcomes. The result of this evaluation 

can provide a better understanding of the theoretical perspectives for an effective professional development 

programme for teachers, designed to support the implementation of a multidisciplinary science module. 
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Appendix I 

NLT has a modular structure, integrating elements from physics, chemistry, biology, mathematics, and physical 
geography within its modules. 

The two modules implemented in this research, ‘The hydrogen car’ and ‘The brain and learning 1 , both have the 
context of a practical situation from everyday life. The topic of hydrogen cars 1 often appears in the newspaper. 
Students investigate whether hydrogen is a suitable alternative for fossil fuels. This question addresses concepts 
from different disciplines and therefore its answer must build upon multidisciplinary and even interdisciplinary 
perspectives. Concepts addressed in the module and traditionally belonging to chemistry are for example an 
electrochemical cell, catalysts, and properties of molecules. Concepts addressed in the module and traditionally 
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belonging to geography are for example climate change and depletion of fossil fuels. Also calculation of 
mathematical and physical forces on a car will be dealt with in this module. Through this integrative, 
interdisciplinary character of NLT modules, their content goes beyond being just the sum of the contents of the 
traditional science subjects, although the individual teachers each bring their mono-disciplinary backgrounds. 

Appendix II 

Individual preparation (1) 

Two questions evaluating the individual preparation stage were posed in the interview (Instrument D). First, ‘Is an 
e-mail from the coordinator useful for starting earlier with the preparation of a module?’ Secondly, ‘Is it useful to 
identify the science equipment needed for the module?’ 

Teacher questions from the individual preparation were collected at the beginning of the preparation seminar and 
then incorporated into the seminar programme, to create a strong sense of ownership with the programme. By 
collecting these questions, we observed whether teachers did the individual preparation activity. 

Preparation seminar (2) 

To evaluate the preparation seminar, teachers were given some open questions to solicit reactions about ‘What was 
missing in the preparation seminar?’ (Instruments A and D), and a five-part question ‘How did the preparation 
seminar contribute to ideas: a) for preparing the experiments, b) for the planning and organisation of the module, c) 
for different assessment methods, d) for field trips or guest lecturers, and e) for motivating students?’ (Instrument 
A), 

These questions about the contribution of the preparation seminar were also asked as questions on a Likert scale 
from 1-5 with ‘5’ as excellent (Instrument C). Instrument C included as well the following two extra questions: 
‘Has the preparation seminar contributed to ideas: f) about the chapter order in your planning and g) how to deal 
with the prior knowledge of students?’ 

The ‘chapter order’ refers to the order in which a module can be taught. Normally one would teach first chapter 1 
followed by 2, 3, 4 etc. but in this subject alternatives are available and also logical: for instance teach the 
practical chapter concurrent with the theoretical chapter, or bring the chapter with the students’ own research 
more to the front. 

Some more open questions were posed immediately after the preparation seminar (Instrument A); ‘How useful 
was attending the preparation seminar?’, ‘How has your understanding of teaching changed after the preparation 
seminar?’, and ‘How has your self-confidence changed after the preparation seminar?’ 

Online support (3) 

The coordinator of the professional development programme organised regular online support among all teachers 
by asking the teachers every Friday to describe their individual good practices and problematic experiences with 
the module for that week in an e-mail; they could also post requests for assistance or guidance on aspects of the 
module. 

After teaching the module, the teachers were asked how useful the Friday e-mail exchange was and whether they 
would prefer a different medium of exchange (Instrument C). 

Reflection meeting (4) 

In the final interview, three open questions about the reflection meeting were posed to the teachers: ‘How do you 
assess the collaborative reflection meeting?’, ‘What are useful elements from discussing the module during the 
evaluation meeting?’, and ‘What are useful elements from discussing contributing issues?’ (Instrument D). 

During the professional development programme teachers collaborated in different ways with colleagues from 
different schools. The teachers were asked after teaching and in the final interview: ‘Do you assess the 
collaboration with colleagues from different schools positively or negatively?’, followed by asking ‘What was 
positive about the collaboration with colleagues from other schools?’ (Instruments C and D). 
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